Neural Integration of Multimodal Events

Our actions produce physical events that give rise to perceptual information across multiple modalities. A trip to the beach can be identified through all five of our senses: seeing the waves roll in from the horizon, hearing the waves break on the sand, smelling the ocean life, feeling the cool sensation of the water on our skin, and tasting the salty water. Yet information from one perceptual modality is not experienced separately from information in another modality; instead, our perception of the ocean blends the modality-specific information from our senses into an integrated, coherent percept. In a series of three experiments, we address how such percepts come to be; that is, what cues in the signal are used to combine different sensory information and, concurrently, what are the neural processing mechanisms sensitive to these integration cues. Multimodal interactions have been found in behavioral and neural research. In both cases, multimodal stimuli generate responses that are not the simple summation of their unimodal counterparts. This finding led researchers to examine what stimulus properties enable the efficient integration of information across modalities, and this research has identified three integration cues – time, space, and semantics – that influence the process of integration. In addition, research with both animals and humans reveal a consistent network of brain regions in posterior, temporal, and frontal cortices recruited during multimodal processing. However, the functional role of the regions within the extended multimodal network is relatively unknown. Consequently, our studies independently manipulate temporal and semantic congruency using a novel stimulus set of environmental events. The studies ask whether the two integration cues share the same neural resources or recruit separable neural substrates. We also explore how these two integration cues interact with one another, and in particular, whether the timecourse of one cue differs from the other. That is, does a low-level cue like temporal synchrony that is carried in the perceptual signals themselves influence multimodal processing earlier than a higher-order cue like semantic congruency that relies on stored, experiential knowledge. In short, the series of three studies aim to unravel the neural basis of multimodal integration. Behavioral research has highlighted three major characteristics of multimodal interactions. First, the response times for detection or identification of a stimulus are faster for multimodal stimuli than for unimodal stimuli. Second, information from one modality can bias another modality, resulting in perceptual illusions that highlight the role of multimodal integration …